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DETAILED ACTION 

Claim Rejections - 35 USC §102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

2. Claims 1, 2, and 5 are rejected under 35 U.S.C. 102(b) as being anticipated by State et al. 
(NPL Document "Superior Augmented Reality Registration by Integrating Landmark Tracking 
and Magnetic Tracking", herein referred to as "State"). 

As per claim 1, State teaches the claimed: 

1 . A three-dimensional image display method 
comprising: 

detecting a position of a light source existing in real space; 
By teaching of: 

The (real) light source is tracked (by the mechanical 
arm) 

(pg. 437, 3 rd paragraph under section 8) 

(emphasis added to various quotations from the reference) 

Here, the position of the light source is detected through the tracking of the mechanical arm. 

State teaches the claimed: 

comparing the position of the light source and a virtual position of a display 
object in a three-dimensional image displayed in real space to obtain a relative 
positional relation therebetween; 

By teaching of: 

Figure 8 demonstrates a virtual object, a knot, casting a 
shadow on a real object, a sculpture. The geometry of the 
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sculpture was digitized with the mechanical arm and placed in 
the scene. The (real) light source is tracked (by the mechanical 
arm), and the shadow map is calculated in real-time 

(pg. 437, 3 rd paragraph under section .8) 

A tracked light source moves real and virtual shadows in sync, 
(pg. 436, caption under figure 8) 

Here, in order for the light source to cause virtual shadowing behind the three-dimensional image 

(where the displayed three dimensional image is the knot) the positions of both the light source 

and the object would have to be known. This is required in order to form the correct virtual 

shadows in response to the light cast from the light source and drawing the virtual shadowing 

caused by the three dimensional object (also see figure 8 where this is shown). 

State teaches the claimed: 

shading in the three-dimensional image 

By teaching of: 

Figure 9 shows a similar scene. The knot intersects the real 
objects, emphasizing the accurate registration of the synthetic 
imagery (the knot and its shadow) with the real cuboids, 
(pg. 437, 3 rd paragraph under section 8) 

Here, figure 9 shows shading on the three-dimensional image itself (where the three dimensional 

image is the knot). 



As per claim 2, State teaches the claimed: 

2. The method according to claim 1, further comprising: 
detecting lightness of the light source. 

By teaching of: 

CCD video cameras . . . attached to the HMD 
(pg. 430, 1st paragraph under section 4) 
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The second component of the landmark finder is the image 
analyzer, which starts its search for a landmark by inspecting 
the search area defined by the landmark predictor, 
(pg. 432, 1 st paragraph under section 6.2.2) 

For our specific camera and frame grabber hardware, and under 
the lighting conditions in our lab, such a simple algorithm can 
reliably distinguish between only a small number of different 
colors 

(pg. 432, 2nd paragraph under section 6.2.2) 
Here, by inspecting a search area using an image analyzer, the lightness is detected as indicated 
in the reference where different colors are detected. Since some colors appear darker or lighter 
than others the lightness is detected in this analysis. 



As per claim 5, this claim is similar in scope to claim 1, and is rejected under the same rationale. 



Claim Rejections - 35 USC §103 

3. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 6-10 are rejected under 35 U.S.C. 103(a) as being unpatentable over State. 



As per claim 6, the reasons and rationale for the rejection of claim 1 is incorporated herein. 
State does not explicitly teach the claimed: 
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a plurality of detectors which detects a position of a light source existing in real 
space; 

However, State suggests the claimed limitation by teaching of in figure 6 of two detectors 
(cameras labeled cl and c2) used to calculate a position of a landmark. It would have been 
obvious to one of ordinary skill in the art to incorporated this position detection method using 
multiple detectors to detect a position of a light source as well. The modification can be 
achieved by replacing the landmark labeled (L2) in figure 6 with a light source, and using 
detectors cl and c2 to detect the light source position. One advantage to such a modification is 
to allow for a more freely controlled light source, which is not constrained by a mechanical arm 
for position detection, by utilizing a plurality of detectors to detect the light source position. 

As per claims 7 and 8, State teaches the claimed: 

a display surface configured to display the three-dimensional image 

By teaching of: 

Head-Mounted Displays (HMDs) 
(pg. 429, 1 st paragraph under section 1) 

Here, the head-mounted display has an associated display surface or surfaces near the eyes that 

produce the three-dimensional image for the user to view. 

State does not explicitly teach the claimed: 

the detector is disposed on at least one of the display surface and a surface 
adjacent to the display surface. 

the detector is disposed to be adjacent to the display surface. 
However, States suggests the claimed limitations by teaching of: 
CCD video cameras . . . attached to the HMD 
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(pg. 430, 1st paragraph under section 4) 
Here, the video cameras are located adjacent to the display and are capable of detecting of light. 
It would have been obvious to one of ordinary skill in the art to achieve the claimed limitation by 
utilizing the CCD video cameras as light position detectors which are adjacent to the display 
surface. The reference suggests this use of the CCD video cameras for light position detection in 
figure 6 where cameras CI and C2 are used to detect the position of landmark L2. The 
motivation of claim 6 is incorporated herein. 

As per claim 9, State does not explicitly teach the claimed: 

9. The device according to claim 5, wherein the detector is disposed at a position 
where the detector detects the 

light emitted from the light source located in the same direction as at least one of 
a display direction of the three dimensional image and a direction in which the 
three-dimensional image is observed 

However, State suggests the claimed limitation by teaching of: 

Head-Mounted Displays (HMDs) 
(pg. 429, 1 st paragraph under section 1) 

CCD video cameras . . . attached to the HMD 
(pg. 430, 1st paragraph under section 4) 

where the HMD display displays the three-dimensional image in a direction in front of the user's 

eyes. It would have been obvious to one of ordinary skill in the art to achieve the claimed 

limitations by using the camera as a light position detector instead of the mechanical arm to 

achieve the claimed limitations. The modification can be achieved by using the two cameras 

(detectors) mounted on the head mounted display to detect the position of the light source. The 

result of this modification is that the light will be detected in the same direction as the display 
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direction of the three-dimensional image since the head mounted display displays the image and 
because the cameras (detectors) are mounted on the head mounted display. The motivation of 
claim 6 is incorporated herein. 

As per claim 10, State does not explicitly teach the claimed: 
10. The device according to claim 5, wherein: 

the detector includes three-primary colors detection unit that adds colors to the 
shade. 

State suggests the claimed limitation by teaching of: 

RGB component values . . . distinguish between only a small number of different 
colors 

(pg. 432, 2 nd paragraph under section 6.2.2) 
Further, State suggests of adding colors to the shade in figure 9 where a shadow is added over 
top of the yellow colored circular mark and the colors are mixed. Thus, it would have been 
obvious to one of ordinary skill in the art to achieve the claimed limitations by using the camera 
as a light position detector instead of the mechanical arm. The modification can be achieved by 
using the two cameras (detectors) mounted on the head mounted display to detect the position of 
the light source and thus also detect the three-primary colors. The modification is possible 
because the reference states the cameras are capable of detecting three-primary colors. The 
motivation of claim 6 is incorporated herein. 



Application/Control Number: 10/612,009 



Page 8 



Art Unit: 2628 

5. Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over State in 
view of Drettakis et al. (NPL Document "Interactive Common Illumination for Computer 
Augmented Reality", herein referred to as "Drettakis"). 



As per claim 3, the reasons and rationale for the rejection of claim 1 is incorporated herein. 

State does not explicitly teach the claimed: 

detecting positions of a plurality of light sources existing in real space; 

Drettakis teaches the claimed limitation by teaching of: 

We have introduced a new framework for dealing with the problem of common 
illumination between real and synthetic objects and light sources in the context of 
computer augmented reality. 
(1 st paragraph under section 6.3) 

We have briefly discussed some possible future research paths, by removing the 
restrictions one by one, to achieve interactive common illumination for first a 
moving camera, then moving lights and finally moving real objects. 
(3rd paragraph under section 6.3) 

Here, in order to achieve interactive illumination with a CG object the positions of the plurality 

of light sources must be known in order to correctly add the realistic shadows effects. Without 

knowing the positions of the light sources the shadow effects can look unrealistic. Thus, the 

system detects a plurality of light source positions. 

It would have been obvious to one of ordinary skill in the art at the time of invention to 

combine State and Drettakis. Drettakis teaches one advantage of the combination by teaching of: 

Novel advances in computer vision are used for camera 
calibration and user-friendly modeling of the real scene, a recent interactive 
radiosity update algorithm is adapted to provide fast illumination update and 
finally textured polygons are used for display. This approach allows interactive 
update rates on mid-range graphics workstations. Our new framework will 
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hopefully lead to CAR systems with interactive common illumination without 
restrictions on the movement of real or synthetic objects, lights and cameras, 
(abstract) 

where State would benefit from the added functionality. 



As per claim 4, State does not explicitly teach the claimed: 

4. The method according to claim 3, further 
comprising: 

obtaining a position of a single virtual light source, which represents the plurality 
of light sources; and 

Drettakis teaches the claimed limitation by teaching of: 

In the system presented we have not shown the addition of virtual lights. This is 
not too hard to achieve, but requires some modification to the incremental update 
approach, since the addition of a light source typically affects a large part of 
the environment. In addition, special attention must be taken in the re-scaling of 
the image before display since the addition of a source can add an order (or 
orders) of magnitude to the radiosity values of the scene 
(col 4, lines 10-12) 

Here, the single virtual light source can be represented as the accumulation of radiosity. The 

reference refers to "adding" light onto the total radiosity for each light source. When every light 

source is considered, one can represent this radiosity as a single virtual light source through the 

addition of all individual light sources. 

State does not teach the claimed: 

comparing the position of the virtual light source and the virtual position of the 
display object in the three-dimensional image to obtain the relative positional 
relations therebetween. 

Drettakis teaches the claimed limitation in figures 3 and 4. Figure 4 shows the result of the 

comparison between the virtual light source (as calculated as the accumulated radiosity in figure 
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3) and the virtual position of the display object (the floating box above the desk). The 
comparison of the positions and relative positional relationship result in the rendering of a soft 
and varying shadow on the desk below the floating object. The varying nature of this shadow 
can be due to the plurality of light directional components provide by the virtual light source. 
One advantage to using this claimed feature with State is to achieve realistic soft shadows affects 
utilizing multiple light sources. 

Response to Arguments 
1 . Applicant's arguments filed 2/27/2007 have been fully considered but they are not 
persuasive. 

Applicant argues State fails to teach the claimed comparing the position of the light source and a 
virtual position of a display object in a three-dimensional image displayed in real space (middle 
of page 3 in filed response) and argues the sculpture cannot constitute a three dimensional image 
displayed in real space because the sculpture is not an image and that is not displayed (top of 
page 4 in filed response) 

The examiner respectfully maintains that the rejections are proper because in figure 8 on page 
436 of State shows a virtual object that is displayed in real space, the knot. In this figure, the 
sculpture is an actual real object and thus is not displayed. The examiner is not intending to rely 
upon the sculpture as a displayed image. Further, in this figure, the knot is floating in mid air to 
emphasis that it is a virtual object, which is displayed through the head mounted display (HMD). 
Further, the knot casts a shadow on real objects in the scene and has shading on itself based upon 
a real light source (the actual light is not shown but would be located to the left of the scene). 
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Further, since the shadows cast are a displayed virtual object, the shadows themselves can also 
be considered virtual or displayed images. In this instance, the virtual objects such as the knot are 
displayed on the HMD display, and yet to the user appear as displayed in real space. The claim 
language requires "a virtual position of a display object in a three-dimensional image displayed 
in real space". This phrase is taught by State where the user wearing a HMD with see a virtual 
position of the floating knot in three-dimensional image as displayed in real space according to 
the output shown in figure 8. 

Applicant further argues that the objects shaded in the HMD are not objects shaded in real space 
because the HMD provides shading only to objects as displayed on the HMD (middle of page 4 
in filed response). 

The examiner respectfully maintains that the rejections are proper because based upon the output 
in figure 8 the user would view the output shaded as shading appearing in real space. The user 
wearing the HMD display gives the illusion that the floating knot would create a shadow and 
thus would appear to be shading in real space. For example, with the tracking technology, the 
user wearing the HMD could move about the area and interact with objects in real space and still 
see the displayed shading created by the knot. Thus, under this interpretation of the claim 
language the shading and display object can be both a display object on the three-dimensional 
image (as claimed in claim 1, for example) and "displayed in real space" (also claimed in claim 
1) because the HMD makes this displaying possible. 
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Conclusion 

2. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571) 272-7642. 
The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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